Complex software systems, such as Enterprise Resource Planning (ERP) systems, have found their way into practically all medium and large-scale businesses, and have become the computational backbone for conducting many aspects of business. Thus, it is important these the software systems perform as expected. Consequently, organizations often devote many resources for the purpose of testing and validating the performance of their software systems. Every time there are changes to a system, such as when a software module is installed, removed, updated, and/or customized, tests should typically be run in order to determine that the software system operates as expected and that there were no adverse effects. In particular, implementing configuration changes to software systems, e.g., changes to values in configuration tables or changes to user generated code, are often considered to be a risky process which requires regression testing to validate affected configuration elements (e.g., business processes impacted by the changes). Often, the cost and time of a full regression test may be overwhelming.
Furthermore, deciding which tests to run, and what those tests should involve, in order to validate a configuration change is typically not an easy task. Often, configuration changes may have unintended and/or unexpected side effects, which a tester may not be aware of. Since tests are usually designed and run by each organization separately on the organization's systems, this means that each organization must build up the necessary knowledge, and independently discover important aspects that should be tested when validating different configuration changes. Gaining this knowledge may require much effort and experience; in the meantime, testing the software systems may be a less effective and prolonged process. However, organizations that are able to utilize each other's testing-related knowledge, which is in a sense a wisdom of the crowd (of testers), may be able to come up with a more effective and efficient testing plan.
Collecting testing data from multiple organizations may not be that helpful if there is no way to sift through the massive amounts of data that can be collected and select appropriate tests for the task at hand. Additionally, not all tests are equal. While some organizations may have knowledge of how to test certain aspects of a software module (e.g., effective tests for testing a certain configuration change), others may have devised erroneous, ineffective, and/or irrelevant tests.